Generalization Error of Linear Neural Networks in an Empirical Bayes Approach
نویسندگان
چکیده
It is well known that in unidentifiable models, the Bayes estimation has the advantage of generalization performance to the maximum likelihood estimation. However, accurate approximation of the posterior distribution requires huge computational costs. In this paper, we consider an empirical Bayes approach where a part of the parameters are regarded as hyperparameters, which we call a subspace Bayes approach, and theoretically analyze the generalization error of three-layer linear neural networks. We show that a subspace Bayes approach is asymptotically equivalent to a positivepart James-Stein type shrinkage estimation, and behaves similarly to the Bayes estimation in typical cases.
منابع مشابه
Generalization Performance of Subspace Bayes Approach in Linear Neural Networks
In unidentifiable models, the Bayes estimation has the advantage of generalization performance over the maximum likelihood estimation. However, accurate approximation of the posterior distribution requires huge computational costs. In this paper, we consider an alternative approximation method, which we call a subspace Bayes approach. A subspace Bayes approach is an empirical Bayes approach whe...
متن کاملVariational Bayes Solution of Linear Neural Networks and Its Generalization Performance
It is well known that in unidentifiable models, the Bayes estimation provides much better generalization performance than the maximum likelihood (ML) estimation. However, its accurate approximation by Markov chain Monte Carlo methods requires huge computational costs. As an alternative, a tractable approximation method, called the variational Bayes (VB) approach, has recently been proposed and ...
متن کاملPrediction of monthly rainfall using artificial neural network mixture approach, Case Study: Torbat-e Heydariyeh
Rainfall is one of the most important elements of water cycle used in evaluating climate conditions of each region. Long-term forecast of rainfall for arid and semi-arid regions is very important for managing and planning of water resources. To forecast appropriately, accurate data regarding humidity, temperature, pressure, wind speed etc. is required.This article is analytical and its database...
متن کاملInvariant Empirical Bayes Confidence Interval for Mean Vector of Normal Distribution and its Generalization for Exponential Family
Based on a given Bayesian model of multivariate normal with known variance matrix we will find an empirical Bayes confidence interval for the mean vector components which have normal distribution. We will find this empirical Bayes confidence interval as a conditional form on ancillary statistic. In both cases (i.e. conditional and unconditional empirical Bayes confidence interval), the empiri...
متن کاملAN EXTENDED FUZZY ARTIFICIAL NEURAL NETWORKS MODEL FOR TIME SERIES FORECASTING
Improving time series forecastingaccuracy is an important yet often difficult task.Both theoretical and empirical findings haveindicated that integration of several models is an effectiveway to improve predictive performance, especiallywhen the models in combination are quite different. In this paper,a model of the hybrid artificial neural networks andfuzzy model is proposed for time series for...
متن کامل